A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.
This invention relates to multiprocessing data processing systems, and more particularly to symmetrical multiprocessor data processing systems that use either a clustered or flat multiprocessor architecture. More specifically, the present invention relates to a software method for routing interrupts within a multiprocessor system.
Systems having coordinated multi-processors were first developed and used in the context of mainframe computer systems. More recently, however, interest in multiprocessor systems has increased because of the relatively low cost and high performance of many microprocessors, with the objective of replicating mainframe performance through the parallel use of multiple microprocessors.
A variety of architectures have been developed including a symmetrical multiprocessing (xe2x80x9cSMPxe2x80x9d) architecture, which is used in many of today""s workstation and server markets. In SMP systems, the processors have symmetrical access to all system resources such as memory, mass storage and I/O.
The operating system typically handles the assignment and coordination of tasks between the processors. Preferably, the operating system distributes the workload relatively evenly among all available processors. Accordingly, the performance of many SMP systems may increase, at least theoretically, as more processor units are added. This highly sought-after design goal is called scalability.
One of the most significant design challenges in many multiprocessor systems is the routing and processing of interrupts. An interrupt may generally be described as an event that indicates that a certain condition exists somewhere in the system that requires the attention of at least one processor. The action taken by a processor in response to an interrupt is commonly referred to as the xe2x80x9cservicingxe2x80x9d or xe2x80x9chandlingxe2x80x9d of the interrupt.
Intel Corporation published a Multiprocessor (MP) specification (version 1.4) outlining the basic architecture of a standard multiprocessor system that uses Intel brand processors. Complying with the Intel Multiprocessor (MP) specification may be desirable, particularly when using Intel brand processors. According to the Intel Multiprocessor (MP) Specification (version 1.4), interrupts are routed using one or more Intel Advanced Programmable Interrupt Controllers (APIC). The APICs are configured into a distributed interrupt control architecture, as described above, where the interrupt control function is distributed between a number of local APIC and I/O APIC units. The local and I/O APIC units communicate over a bus called an Interrupt Controller Communications (ICC) bus. There is one local APIC per processor and, depending on the total number of interrupt lines in an Intel MP compliant system, one or more I/O APICs. The APICs may be discrete components separate from the processors, or integrated within the processors.
The destination of an interrupt can be one, all, or a subset of the processors in the Intel MP compliant system, hereinafter the interrupt xe2x80x9cAffinityxe2x80x9d. The sender specifies the destination of an interrupt in one of two destination modes: physical destination mode or logical destination mode. In physical destination mode, the destination processor is identified by a local APIC ID. The local APIC ID is then compared to the local APIC""s actual physical ID, which is stored in a local APIC ID register within the local APIC. The local APIC ID register is loaded at power up by sampling configuration data that is driven onto pins of the processor. For the Intel P6 family of processors, pins A11# and A12# and pins BRO# through BR3# are sampled. Up to 15 local APICs can be individually addressed in the physical destination mode.
The logical destination mode can be used to increase the number of APICs that can be individually addressed by the system. In the logical destination mode, message destinations are identified using an 8-bit message destination address (MDA). The MDA is compared against the 8-bit logical APIC ID field of the APIC logical destination register (LDR).
A Destination Format Register (DFR) is used to define the interpretation of the logical destination information. The DFR register can be programmed for a flat model or a cluster model interrupt delivery mode. In the flat model delivery mode, bits 28 through 31 of the DFR are programmed to 1111. The MDA is then interpreted as a decoded address. This delivery mode allows the specification of arbitrary groups of local APICs by simply setting each APIC""s corresponding bit to 1 in the corresponding LDR. Broadcast to all APICs is achieved by setting all 8 bits of the MDA to one. As can be seen, the flat model only allows up to 8 local APICs to coexist in the system.
Unisys Corporation, assignee hereof, manufactures a multi-processor system like that described above under the model name ES7000. This system contains 96 PCI slots and up to 32 processors. To perform a function, a device connected to a PCI slot must send an interrupt vector to one of the processors. The code invoked by the interrupt vector must specifically instruct the processor what function to perform.
One approach for specifying the function the processor will perform is assigning a unique vector to every device needing an interrupt. Each peripheral will send a unique interrupt vector identifying the function. Each processor generally has a table mapping an interrupt vector to a particular function. In this case, the table for each processor will be identical.
As an example, the first peripheral can send an interrupt vector labeled xe2x80x9851,xe2x80x99 the second peripheral xe2x80x9852,xe2x80x99 the third peripheral xe2x80x9853,xe2x80x99 and so on. Because the table for each processor is identical, the functions associated with interrupt vectors xe2x80x9851,xe2x80x99 xe2x80x9852,xe2x80x99 and xe2x80x9853xe2x80x99 can correctly be referenced from any of the processors. Peripherals are not limited to using a single interrupt vector. In this example, the first peripheral could have sent vectors xe2x80x9850xe2x80x99 and xe2x80x9851,xe2x80x99 each referencing a different function call.
This approach can be expanded to use lowest priority routing, where the processor that is at the lowest Interrupt Request Level (IRQL), runs the Interrupt Service Routine (ISR) for the current interrupt. This feature improves performance by distributing interrupts in an efficient manner.
This approach, however, has one major drawback. The number of unique interrupts assigned to peripherals is limited to the total number of interrupt vectors that can be created. If the number of required vectors exceeds the number of available unique interrupt vectors, the peripherals which failed to obtain interrupt vectors will be inoperable. One solution to this involves sharing the physical interrupt line at the hardware level (The PCI specification amongst others defines a method to do this). The downside to this is that it results in extra overhead, typically in the operating system to establish which of the peripheral devices sharing the hardware interrupt line is requesting the interrupt. In a high performance system, the overhead involved in doing this may not be acceptable. Another solution, the subject of this application, is to break the limitation that all processors must have identical mapping tables, thereby significantly increasing the vector pool available in a multi-processor system.
Accordingly, by affinitizing each interrupt to a specific processor, unique interrupt vectors are no longer required. Instead, the same interrupt vector is sent to each processor, but each processor can point to a different interrupt subroutine (or xe2x80x9cISRxe2x80x9d), allowing each interrupt vector to be interpreted differently depending upon which processor it is sent to. Additionally, the tables associated with each processor are no longer identical. Each processor may have a different function referenced by a particular interrupt vector. Because of the affinity between processor and peripheral, peripheral devices can still reference the desired function.
As an example, consider a system with a total of four processors and at least four peripherals. Processor one can be associated with peripheral two, processor two with peripheral one, processor three with peripheral three, and processor four with peripheral four. Peripherals one through four would send the same interrupt vector to their corresponding processors. Peripherals five and above would send different interrupt vectors for subsequent function requests.
Lowest priority routing allows more than one bit set in the destination set. The disclosed embodiment requires one and only one bit set in the destination set. Moreover, the disclosed method enables an enterprise server to fully populate its available I/O slots by allowing the complete vector pool to be available to each processor. This expands the number of vectors by the number of processors in the configuration. If two processors are available, then the number of interrupt vectors is effectively doubled. If thirty-two processors are available then there are thirty-two times as many interrupt vectors available. In certain configurations, this allows for a higher performance system, by removing the need to share the same interrupt line at the hardware level in the system.
These and other objects, which will become apparent as the invention is described in detail below, are provided by a set of data structures and algorithms which determine the allocation scheme best suited for optimizing a particular facet of vector allocation. The invention is a method for assigning a multiplicity of interrupt vectors in a symmetric multi-processor computing environment, wherein a multiplicity of peripherals are coupled to the multi-processor computer. The disclosed method assigns interrupt vectors from the multiplicity of peripherals, and includes the steps of determining if an interrupt resource from a given peripheral has already been assigned, and if not; assigning an affinity and vector to this interrupt resource. Moreover, a determination is made if the affinity and vector assigned in the previous step are within the operational characteristics of the multi-processor computer system.
It is an object of the present invention to provide greater peripheral connectivity in a multi-processor system.
Another object of the present invention is to provide verification of operation within the operational characteristics of a given computing platform.
An advantage of the present invention is the ability to connect a greater number of peripherals to a given multi-processor system without the need to share interrupts at the hardware level.
Another advantage of the present invention is to improve the performance of the computing platform in certain applications.
Still other objects and advantages of the present invention will become readily apparent to those skilled in the art from the following detailed description, wherein is shown and described only the preferred embodiment of the invention, simply by way of illustration of the best mode contemplated of carrying out the invention. As will be realized, the invention is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the invention. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive, and what is intended to be protected by Letters Patent is set forth in the appended claims. The present invention will become apparent when taken in conjunction with the following description and attached drawings, wherein like characters indicate like parts, and which drawings form a part of this application.